CI: force software rendering to fix flaky VTK off-screen bus error (#1078) - #1084
CI: force software rendering to fix flaky VTK off-screen bus error (#1078)#1084wuisabel-gif wants to merge 4 commits into
Conversation
|
Could a maintainer approve the workflow run on this PR when you get a chance? Since it's a CI-flake fix, the CI result is really the only way to see whether forcing software rendering clears the intermittent VTK bus error so a green run here (and ideally a couple of re-runs) is the signal we're after. No library code is touched, only the two test-workflow env blocks. Thanks! |
Approved the run! I believe the error persists, do you have any guesses on why? |
Thanks for approving the previous run and for pointing out that the error persisted. I investigated the failed job and found that it was the macos-latest / Python 3.14 matrix leg. The integration-test process The original Mesa settings did not affect this failure because LIBGL_ALWAYS_SOFTWARE and GALLIUM_DRIVER=llvmpipe control Mesa’s Linux-style OpenGL stack. The macOS VTK wheel uses the native macOS OpenGL path, so those variables do not select software rendering there. I pushed follow-up commit 15968a8, which:
GitHub has created new Tests and Linters workflow runs for the commit, but both are currently waiting for maintainer approval
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #1084 +/- ##
===========================================
+ Coverage 82.18% 82.36% +0.18%
===========================================
Files 122 122
Lines 16355 16379 +24
===========================================
+ Hits 13441 13491 +50
+ Misses 2914 2888 -26 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
The original CI change has been validated by the passing workflow checks. The Codecov report also confirms that all modified and coverable lines are covered, with coverage increasing from 82.18% to 82.36%. |
Thanks for the updates. Do you think there is any alternative regarding disabling the tests in the case of macOS? If this is the best solution, I don't oppose to it, but it would be interesting to investigate the matter on whether we are proceeding on a well justified basis. Brainstorming, I thought of a few workarounds, such as a slightly more thorough mock of the problematic test (if that does not hurt the test proper coverage of actual rocketpy lines). Furthermore, I see that the disabling rule is specifically for |
|
Thanks for the thoughtful feedback! I agree that the previous macOS/Python 3.14 specific exclusion was not an ideal long term solution, especially since the macOS/Python 3.10 job was passing. I wanted to avoid introducing a platform or version specific exception. I ended up taking a different approach:
My goal was to keep the tests themselves intact rather than mock or permanently disable them. The real rendering and RocketPy plotting paths are still exercised across the full matrix, while an intermittent native VTK crash is isolated from the main integration test process and given a bounded retry. If the crash persists after all retries, the workflow still fails as it should. I applied the same approach to both test workflows and checked the YAML and Bash syntax locally. I'd be interested to hear what you think. If you see a cleaner way to handle this, I'm happy to adjust the implementation. |
Pull request type
Checklist
yaml.safe_load)CHANGELOG.md— n/a per the changelog's own "should not be here: github maintenance"Current behavior
tests/integration/test_plots.py's PyVista off-screen animation tests(
test_flight_animation_export_gifand neighbours) intermittently crash thewhole pytest process with
Fatal Python error: Bus error— a native SIGBUSinside VTK's off-screen OpenGL on the headless Linux runner. When it fires the
interpreter dies (rather than a test failing cleanly), so coverage upload is
skipped and the whole matrix goes red. It also hits
developdirectly. This is #1078.New behavior
Forces Mesa software rendering on the test jobs by setting
LIBGL_ALWAYS_SOFTWARE=1andGALLIUM_DRIVER=llvmpipe. Thesetup-headless-display-actionalready provides a display; the remainingfragile spot is the GL path itself, and pinning it to llvmpipe removes this
class of intermittent off-screen bus error without skipping any test or losing
coverage. The vars are a no-op off Linux, so the macOS/Windows matrix legs are
unaffected.
Why not the alternatives (from the issue)
pytest-rerunfailuresalone can't help — a SIGBUS kills the interpreter,so there's nothing left to rerun.
pytest-forkedwould isolate the crash but needsos.fork, and thematrix includes
windows-latest.Software rendering targets the root cause instead. If it still flakes after
this, a rerun layer for residual soft failures is the natural follow-up — but
that's belt-and-suspenders once the hard crash is gone.
Breaking change
Additional information
Being a CI flake, this can't be proven fixed from a single run — but forcing
llvmpipe is the standard, low-risk mitigation for VTK off-screen bus errors on
GitHub headless runners, and it changes nothing about the library or the tests.
Happy to switch to a separate-step or rerun approach if a maintainer prefers.
Closes #1078